Dynamic product information during barcode scanning

ABSTRACT

A computer-implemented method includes receiving a request to scan a barcode and in response, displaying a split screen having a camera image portion and a product information portion. At least part of an image currently being captured by a camera and containing a barcode is displayed in the camera image portion. Product information for a product represented by the barcode is displayed in the product information portion.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a continuation of and claims priority of U.S.patent application Ser. No. 16/031,430, filed Jul. 10, 2018, the contentof which is hereby incorporated by reference in its entirety.

BACKGROUND

Mobile devices such as Smart Phones have become sophisticated computingdevices that include processors capable of executing one or moreapplications, touch screens providing interactive user interfaces andcameras capable of capturing still images and video. Nearly all mobiledevices include a web browser application that allows users to downloadand view pages from websites on the Internet as well as various types ofmedia including documents and videos. Many web browser applicationssupport script execution within the web browser, such as the executionof Java Script.

Mobile devices are typically shipped with a small collection ofapplications that can be executed on the mobile device. Additionally,users are able to download applications on the mobile device through anetwork connection to an application marketplace. However, downloadingadditional applications requires the user to take several steps that actas a barrier to acquiring the functionality provided by theapplications. In particular, the user must discover that the applicationexists either through word of mouth or by performing searches in theapplication marketplace. Additionally, the user must select theapplication from the marketplace and wait for the application todownload to the mobile device. Given the large number of applications inthe marketplace, most users do not download a significant percentage ofthe available applications. As a result, adoption of mobile deviceapplications is typically quite low.

The discussion above is merely provided for general backgroundinformation and is not intended to be used as an aid in determining thescope of the claimed subject matter. The claimed subject matter is notlimited to implementations that solve any or all disadvantages noted inthe background.

SUMMARY

A computer-implemented method includes receiving a request to scan abarcode and in response, displaying a split screen having a camera imageportion and a product information portion. At least part of an imagecurrently being captured by a camera and containing a barcode isdisplayed in the camera image portion. Product information for a productrepresented by the barcode is displayed in the product informationportion.

In accordance with a further embodiment, a mobile device includes acamera, a display, a memory containing instructions, and a processorexecuting the instructions in the memory to perform steps. The stepsinclude receiving a request to scan for barcodes and displaying a userinterface with an image area containing images produced by the camera,and a product area containing product information for a productassociated with a barcode shown in the image area wherein the productinformation changes automatically when the barcode shown in the imagearea changes.

In accordance with a still further embodiment, a method includesexecuting a web application within a browser application on a devicesuch that the web application performs a test to determine if thebrowser application can request images from a camera on the device. Whenthe browser application can request images from the camera, the webapplication displays a control to request barcode scanning and receivesan indication that the control has been used to request barcodescanning. In response, the web application shows a split screen userinterface depicting images captured by the camera in one portion of theuser interface.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 provides a block diagram of a system used in accordance with oneembodiment.

FIG. 2 provides a flow diagram of a method for displaying productinformation using barcodes in a retail environment.

FIG. 3 provides an example user interface on a mobile device provided bya web browser application in accordance with one embodiment.

FIG. 4 provides an example user interface provided by a web browserapplication showing a scan barcode control in accordance with oneembodiment.

FIG. 5 provides an example user interface provided by a web browserapplication showing a split screen in accordance with one embodiment.

FIG. 6 provides an example user interface provided by a web browserapplication showing product information added to the split screen for abarcode in accordance with one embodiment.

FIG. 7 shows a change in the product information for a change in thebarcode shown in the user interface provided by the web browserapplication in accordance with one embodiment.

FIG. 8 provides a flow diagram of a method for displaying productinformation based on a barcode captured by a camera in accordance withone embodiment.

FIG. 9 provides a flow diagram of a method for processing a barcode inaccordance with one embodiment.

FIG. 10 provides a flow diagram of a method of processing selection ofan add-to-cart control displayed with product information in a splitscreen in accordance with one embodiment.

FIG. 11 provides a flow diagram of a method of processing selection of aview-details control displayed with product information in a splitscreen in accordance with one embodiment.

FIG. 12 is a flow diagram of a method performed when a mobile devicereconnects to a network in accordance with one embodiment.

DETAILED DESCRIPTION

The cameras on mobile devices can be used to capture images of barcodeson products. Numbers and/or letters represented by the barcode can thenbe decoded from the barcode and can be submitted as an identifier forthe product. Such identifiers can be submitted as a request to purchasethe product online and/or to receive additional information about theproduct online. In the past, as soon as the barcode was decoded, theuser interface was changed to either take the user to a purchase page orto take the user to a page displaying information about the product. Ifthe user wished to scan another barcode for a different product, theyhad to interact with several controls in order to move away from thepage they have been directed to and bring back a user interface thatcould be used to capture the barcode. These interactions made the mobiledevice inefficient when attempting to receive information for a largenumber of products based on their barcode.

In the various embodiments described below, a user interface is providedin which a split screen is shown with one portion of the screencontaining images provided by a camera in the mobile device containing abarcode of a product and a second portion of the split screen containinginformation for the product associated with the barcode. When the usershifts the mobile device so that the camera captures an image of adifferent barcode, the displayed product information automaticallychanges so that it corresponds to the new barcode without the userhaving to select any controls on the mobile device. As a result, theuser is able to quickly receive product information for a number ofdifferent products simply by scanning over the barcodes of the differentproducts. Further, since the product information changes as soon as thebarcode is capable of being read, the user is able to receive feedbackthat the barcode has been properly positioned within the camera portionof the split screen based on the change in product information in theproduct information portion of the split screen.

“Barcode” as used herein includes one-dimensional or linear codes thatuse black and white vertical lines to encode information as well asmulti-dimensional codes (two or more) including QR codes, that encodeinformation using two-dimensional shapes and/or multiple differentcolors. In some embodiments, the barcode is embedded within anothergraphic. In some embodiments, the barcode can be perceived by human eyesand in other embodiments, the barcode can only be perceived by a cameraor scanner that includes a digital filter.

FIG. 1 provides a block diagram of a system 100 including a mobiledevice 102 that communicates with a website server 106 through a network104. Mobile device 102 includes a touchscreen 108, a processor 110, acamera 112, a microphone 109, a network interface 114, and a memory 116.Processor 110 includes a central processing unit and a local memorycache and is connected to touchscreen 108, camera 112, microphone 109,network interface 114 and memory 116 through one or more buses and/orinterfaces. Processor 110 is capable of executing one or moreapplications, referred to as apps, which can be stored in the localcache or in a combination of the local cache and portions of memory 116during execution. For example, processor 110 is shown to be executingbrowser app 118 in FIG. 1. Browser app 118 is able to issue requests towebsite server 106 through network interface 114 and network 104. Theserequests can include requests to post data to website server 106 and torequest webpages and/or media from website server 106. The requestedwebpages can have embedded therein various types of media, such asdocuments or videos and can further include embedded scripts that can beexecuted by a script engine 122 within browser app 118. For example, inFIG. 1, a script representing a web app 120 is shown to have beendownloaded from website server 106 by browser app 118 so that it can beexecuted by script engine 122. Browser app 118 also includes a cameraAPI 124 that is able to communicate with camera 112 of mobile device 102to activate camera 112 so that it captures a stream of images and sendsthe captured images to camera API 124 for use in the script executed byscript engine 122. Browser app 118 also includes the ability to storestate information for web sessions in the form of cookies 126. In FIG.1, cookies 126 include a shopping cart 128 that indicates the content ofan online shopping cart that includes items that the user has indicatedthey wish to purchase but that have yet to purchase. In otherembodiments, shopping cart 128 is maintained on website server 106.

FIG. 2 provides a flow diagram of a method of using a mobile device toscan barcodes and using the scanned barcodes to display productinformation on the mobile device in accordance with one embodiment. Instep 200 of FIG. 1, a camera interface 150 in web app 120 tests browserapp 118 to determine if browser app 118 implements camera API 124 toallow for access to camera 112. If browser app 118 does not implement acamera API at step 201, web app 120 displays a user interface that doesnot include a scanning button at step 203. However, if camera API 124 issupported by browser app 118, web app 120 presents a scanning button tothe user at step 202 through user interface control 152. In accordancewith one embodiment, the scanning button is presented to the user on asearch screen of web app 120. FIG. 3 shows an example user interface 302generated by web app 120 through browser app 118 that can be used torequest the search screen by selecting search field 300. Upon selectionof search field 300, user interface 400 of FIG. 4 is created by web app120, which includes a scan barcode control 402.

At step 204 of FIG. 2, a request to activate the scanner is receivedwhen a user selects scan barcode control 402. In response, at step 206,camera interface 150 of web app 120 requests a camera image streamthrough camera API 124 of browser app 118. In particular, camera API 124accesses camera 112 to obtain a continuous stream of images captured bycamera 112 and provides that stream of images to camera interface 150 asexecuted by script engine 122.

Camera interface 150 provides the stream of camera images to a UIcontrol 152, which displays a split screen user interface with a cameraimage portion and a product information portion at step 208. Inaccordance with one embodiment, the split screen user interface iscreated within a window created by browser app 110. FIG. 5 provides anexample of the split screen user interface 500 showing the camera imageportion 502 and the product information portion 504. Initially, theproduct information portion is blank since the barcode has yet to bedecoded as shown in FIG. 5. In accordance with some embodiments, thecamera portion includes a barcode target area 506 that provides aclearer view of the camera image than other parts of camera portion 502,which filter the pixels of the camera image. In addition, an instruction508 directs the user to center the barcode within the barcode targetarea 506. At step 210, UI control 152 and barcode processing 154 of webapp 120 continuously evaluate barcodes in the image stream from camera112 to decode information from the barcode and to retrieve and displayproduct information associated with the decoded information. Userinterface 600 of FIG. 6 shows the display of such product information inproduct portion 504. The product information includes a product image602, product price 604, product description 606, a rating 608, an offer610, as well as an ADD TO CART control 612 and a VIEW DETAILS control614 described further below.

The continuous evaluation of barcodes in step 210 means that as camera112 is shifted and a new barcode comes into view, barcode processing 154and UI control 152 automatically update the product informationdisplayed in product information portion 504 without requiring the userto interact with any further controls. For example, user interface 700shows a change in the product information of product information portion504 due to a different barcode 702 being captured by camera 112 as shownin camera portion 502. Thus, the user is able to receive information fora large number of different products simply by moving the camera todifferent barcodes. In addition, the changes in the product informationof product information portion 504 provide feedback to the user that thebarcode has been read by the mobile device.

FIG. 8 provides a flow diagram of a method of performing the continuousevaluation of barcodes in step 210. In step 800 of FIG. 8, barcodeprocessing 154 captures one of the images in the image stream. Inaccordance with one embodiment, step 800 is performed once every 100 ms.At step 802, barcode processing 154 evaluates the snapshot image todetermine if a barcode is present in the image. In accordance with oneembodiment, step 802 involves comparing neighboring pixels to identifyedge pixels where the image transitions from black to white. The numberof such edges is then computed and compared to a threshold. At step 804,if there enough edge transitions in the image to indicate that a barcodeis present, barcode processing 154 processes the image at step 806 toextract the information represented by the barcode such as a sequence ofnumbers and/or letters. The information represented by the barcode isreferred to as barcode data. At step 808, barcode processing 154compares the currently extracted barcode data to the previouslyextracted barcode data to determine if it is the same. If the barcodedata is different from the previously extracted barcode data, web app120 uses web request 156 to request product information/images fromwebsite server 106 for the product associated with the barcode data atstep 810.

If web request 156 successfully retrieves the product information atstep 812, the product information is displayed in the product portion ofthe split screen user interface at step 814 together with an ADD TO CARTbutton and a VIEW DETAILS button. The process of FIG. 8 then returns tostep 800 to capture a new image from the image stream. When an imagedoes not contain a barcode at step 804 or when the barcode data is thesame as the previously extracted barcode data at step 808, UI control152 continues to display the product information/image, ADD TO CARTbutton and VIEW DETAILS button that was previously displayed at step816. Thus, in accordance with such embodiments, the product informationfor the last decoded barcode continues to be displayed until a newbarcode is read at step 808 at which point new product information isrequested at step 810 and is displayed in place of the old productinformation at step 814.

Because web app 120 is executing within browser app 118 and because webapp 120 requires product information to be sent by website server 106,the product information cannot be displayed if mobile device 102 doesnot have a network connection and is therefore considered to be“offline”. If web request 156 fails to receive the product informationat step 812, web request 156 determines if mobile device 102 is offlineat step 818. If mobile device 102 is not offline at step 818, theprocess returns to step 800 to capture a new snapshot from the imagestream. If mobile device 102 is offline at step 818, the barcode data isstored and a listener task is set that will listen for connect event atstep 820. Such a listener will be triggered the next time the mobiledevice connects to the network as discussed further below. At step 822,after the listener has been set, camera interface 150 instructs camera112 to turn off through camera API 124. UI control 152 then displays thepage that was shown before the barcode scan button was selected at step824.

FIG. 9 provides a flow diagram of a method of processing a snapshot toextract barcode data in accordance with one embodiment. At step 900, theimage is divided vertically into 10 parts. At step 902, the pixels alongthe boundaries between each of the 10 parts are converted to grey scalefor contrast. At step 906, the pixels along each line are compared tothe pixel values along the other lines to determine a variance for thestart, middle and end of each line. If each of these variances is small,indicating that the same pixel values are present at each height of theimage, the pixel values are converted into either a one or zero at step904.

At step 906, the bit values determine in step 904 for the barcode arecompared to patterns defined for the particular barcode protocol beingused. The characters or symbols associated with the matching digitalpatterns are then retrieved at step 906.

Returning to FIG. 8, while the split screen user interface is beingdisplayed showing a camera portion and a product portion with an ADD TOCART control and a VIEW DETAILS control, the user can select either theADD TO CART control or the VIEW DETAILS control. When one of thecontrols is selected, the selected button is processed at step 826.

FIG. 10 provides a flow diagram of a method of processing a selection ofan ADD TO CART control such as ADD TO CART 612. At step 1002, an add tocart process 158 adds the product to the electronic shopping cart byeither storing the identity of the product in shopping cart 128 ofcookies 126 or sending a request to website server 106 through networkinterface 114 and network 104 to have the product added to the shoppingcart stored on website server 106 for the user of mobile device 102. Atstep 1004, camera interface 150 uses camera API 124 to instruct camera112 to cease sending images to web app 120. At step 1006, web request156 requests the shopping cart page from website server 106 and at step1008, browser app 118 displays the shopping cart page.

FIG. 11 provides a flow diagram of a method for processing the selectionof a VIEW DETAILS control such as VIEW DETAILS control 614 in step 826.In step 1102 of FIG. 11, camera interface 150 uses camera API 124 toinstruct camera 112 to stop sending images to web app 120. At step 1104,web request 156 requests the product page of the product shown inproduct portion 504 from website server 106. At step 1106, browser app118 displays the product page returned by website server 106.

FIG. 12 provides a flow diagram of a method implemented by a listenerset in step 820 of FIG. 8 when mobile device 102 was offline. In step1200 of FIG. 12, the listener waits for an online connect event. Whenmobile device 102 reconnects to the network at step 1200, processor 110generates an online connect event that causes the listener to retrievethe stored barcode information at step 1202. The listener then requestsproduct information from website server 106 for the barcode data at step1204. At step 1206, the product information is received and at step1208, the listener creates an alert based on the received productinformation, which includes the product information as well as a productimage. An icon for the alert is added to the status bar of the mobiledevice when the alert is created and when the user selects the icon forthe alert at step 1210, browser app 118 is started and a request for theproduct page of the product is made to website server 106 at step 1212.At step 1214, the product page is returned and displayed within browserapp 118.

Although elements have been shown or described as separate embodimentsabove, portions of each embodiment may be combined with all or part ofother embodiments described above.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms for implementing the claims.

What is claimed is:
 1. A computer-implemented method comprising: acamera interface in a web browser application executed by a scriptengine in the web browser application determining whether the webbrowser application supports accessing a camera; the web browserapplication determining whether to display a selectable control in auser interface based on whether the web browser application supportsaccessing the camera, the selectable control for requesting barcodescanning; receiving a request to scan a barcode; in response, the webbrowser application displaying a split screen within a browser window ofthe web browser application, the split screen having a camera imageportion and a product information portion; displaying at least part ofan image currently being captured by a camera and containing a barcodein the camera image portion; and displaying product information for aproduct represented by the barcode in the product information portion.2. The computer-implemented method of claim 1 wherein receiving therequest to scan a barcode comprises receiving the request from the webapplication.
 3. The computer-implemented method of claim 2 when the webbrowser application supports accessing the camera, displaying aselectable control for requesting the scan of the barcode; and whereinreceiving the request to scan the barcode comprises receiving anindication that the selectable control has been selected.
 4. Thecomputer-implemented method of claim 1 further comprising displaying aseries of images captured by the camera in the camera image portion suchthat within the series of images a second barcode is shown in the cameraimage portion and when the second barcode is shown, second productinformation for a product represented by the second barcode isautomatically shown in the product information portion.
 5. Thecomputer-implemented method of claim 1 wherein displaying the productinformation in the product information portion further comprisesdisplaying a control for adding the product to an electronic shoppingcart.
 6. The computer-implemented method of claim 5 wherein displayingthe product information in the product information portion furthercomprises displaying a control for obtaining more information about theproduct.
 7. A method comprising: a camera interface of a web applicationexecuted by a script engine of the web browser application determiningwhether the web browser application supports accessing a camera; the webbrowser application determining whether to display a selectable controlin a first user interface based on whether the web browser applicationsupports accessing the camera, the selectable control for requestingbarcode scanning; receiving a request to scan for barcodes; and the webapplication executed by the script engine of the web browser applicationdisplaying a second user interface within a window of the web browserapplication, the second user interface having an image area containingimages produced by the camera, and an area containing information for anitem associated with a barcode shown in the image area wherein theinformation changes automatically when the barcode shown in the imagearea changes.
 8. The method of claim 7 wherein displaying the areacomprises displaying a control to view details of the item.
 9. Themethod of claim 7 further comprising in response to receiving therequest to scan for barcodes sending a request for images from thecamera through the web browser application.
 10. A method comprising:executing a web application within a browser application on a devicesuch that: the web application performs a test to determine if thebrowser application can request images from a camera on the device; whenthe browser application can request images from the camera, the webapplication displaying a control to request barcode scanning; receivingan indication that the control has been used to request barcodescanning, and in response showing a split screen user interfacedepicting images captured by the camera in one portion of the userinterface.
 11. The method of claim 10 wherein showing a split screencomprises displaying a barcode in the portion of the user interfacedepicting images captured by the camera and displaying information for aproduct associated with the barcode in a second portion of the userinterface.
 12. The method of claim 11 wherein displaying information fora product associated with the barcode comprises displaying informationfor a second product associated with a second barcode automatically whenthe image captured by the camera contains the second barcode instead ofthe barcode.
 13. The method of claim 11 further comprising displaying acontrol for adding the product to an electronic shopping cart with theinformation for the product.
 14. The method of claim 13 furthercomprising displaying a control to request additional information aboutthe product with the information for the product.